Policy plane integration across multiple domains

ABSTRACT

Systems, methods, and computer-readable media for interconnecting SDWANs through segment routing. A first SDWAN and a second SDWAN of a SDWAN fabric can be identified. A segment routing domain that interconnects the first SDWAN and the second SDWAN can be formed across a WAN underlay of the SDWAN fabric. Data transmission between the first SDWAN and the second SDWAN can be controlled by performing segment routing through the segment routing domain formed between the first SDWAN and the second SDWAN.

TECHNICAL FIELD

The subject matter of this disclosure relates in general to the field ofcomputer networking, and more particularly, to systems and methods forinterconnecting SDWANs through segment routing.

BACKGROUND

The enterprise network landscape is continuously evolving. There is agreater demand for mobile and Internet of Things (IoT) device traffic,Software as a Service (SaaS) applications, and cloud adoption. Inaddition, security needs are increasing and certain applications canrequire prioritization and optimization for proper operation. As thiscomplexity grows, there is a push to reduce costs and operating expenseswhile providing for high availability and scale.

Conventional wide area network (WAN) architectures are facing majorchallenges under this evolving landscape. Conventional WAN architecturestypically consist of multiple Multi-Protocol Label Switching (MPLS)transports, or MPLS paired with Internet or Long-Term Evolution (LTE)links used in an active/backup fashion, most often with Internet or SaaStraffic being backhauled to a central data center or regional hub forInternet access. Issues with these architectures can includeinsufficient bandwidth, high bandwidth costs, application downtime, poorSaaS performance, complex operations, complex workflows for cloudconnectivity, long deployment times and policy changes, limitedapplication visibility, and difficulty in securing the network.

In recent years, software-defined enterprise network solutions have beendeveloped to address these challenges. Software-defined enterprisenetworking is part of a broader technology of software-definednetworking (SDN) that includes both software-defined wide area networks(SDWAN) and software-defined local area networks (SDLAN). SDN is acentralized approach to network management which can abstract away theunderlying network infrastructure from its applications. Thisde-coupling of data plane forwarding and control plane can allow anetwork operator to centralize the intelligence of the network andprovide for more network automation, operations simplification, andcentralized provisioning, monitoring, and troubleshooting.Software-defined enterprise networking can apply these principles of SDNto the WAN and a local area network (LAN).

Currently SDWANs can be combined to form a single network, e.g. a verylarge network. For example, regional campus networks can form a verylarge network of one or more entities. Specifically, instead of buildingone large SDWAN, a hierarchy of SDWANs can be created to form a networkby building regional SD-WAN networks/clouds. Often these regional SDWANsare terminated at hub sites, Data Centers and/or colocation facilities.In forming a network through a plurality of SDWANs, facilitatingcommunication between the SDWANs, e.g. interconnecting the SDWANs, iscritical to ensuring that the network functions properly. However,interconnecting separate SDWANs is difficult to accomplish.Specifically, interconnecting separate SDWANs through a SDWAN fabricsupporting the SDWANs is difficult to properly implement. Theretherefore exist needs for systems and methods of interconnectingseparate SDWANs forming a larger network. More specifically, there existneeds for systems and methods of interconnecting separate SDWANs througha SDWAN fabric in which the SDWANs are formed.

BRIEF DESCRIPTION OF THE FIGURES

To provide a more complete understanding of the present disclosure andfeatures and advantages thereof, reference is made to the followingdescription, taken in conjunction with the accompanying drawings, inwhich:

FIG. 1 illustrates an example of a high-level network architecture inaccordance with an embodiment;

FIG. 2 illustrates an example of a network topology in accordance withan embodiment;

FIG. 3 illustrates an example of a diagram showing the operation of aprotocol for managing an overlay network in accordance with anembodiment;

FIG. 4 illustrates an example of a diagram showing the operation ofvirtual private networks for segmenting a network in accordance with anembodiment;

FIG. 5A illustrates a diagram of an example Network Environment, such asa data center;

FIG. 5B illustrates another example of Network Environment;

FIG. 6 shows an example network environment of interconnected SDWANs;

FIG. 7 illustrates an example of a network device; and

FIG. 8 illustrates an example of a bus computing system wherein thecomponents of the system are in electrical communication with each otherusing a bus.

DESCRIPTION OF EXAMPLE EMBODIMENTS

The detailed description set forth below is intended as a description ofvarious configurations of embodiments and is not intended to representthe only configurations in which the subject matter of this disclosurecan be practiced. The appended drawings are incorporated herein andconstitute a part of the detailed description. The detailed descriptionincludes specific details for the purpose of providing a more thoroughunderstanding of the subject matter of this disclosure. However, it willbe clear and apparent that the subject matter of this disclosure is notlimited to the specific details set forth herein and may be practicedwithout these details. In some instances, structures and components areshown in block diagram form in order to avoid obscuring the concepts ofthe subject matter of this disclosure.

Overview

A method can include identifying a first SDWAN and a second SDWAN of aSDWAN fabric. A segment routing domain through the SDWAN fabric can beformed across a WAN underlay that interconnects the first SDWAN and thesecond SDWAN. Data transmission between the first SDWAN and the secondSDWAN can be controlled by performing segment routing through thesegment routing domain formed between the first SDWAN and the secondSDWAN.

In various embodiments, the segment routing domain can be formed throughthe SDWAN fabric by pre-building a plurality of paths through the WANunderlay between the first SDWAN and the second SDWAN. The plurality ofpaths can be selectable to control data transmission between the firstSDWAN and the second SDWAN through the segment routing domain.

In certain embodiments, the plurality of paths can be changeable tocontrol data transmission between the first SDWAN and the second SDWANthrough the segment routing domain.

In various embodiments, performance measurements of links in the WANunderlay forming paths in the segment routing domain can be gathered.Segment routing through the segment routing domain can be controlledbased on the performance measurements of the links in the WAN underlayto control data transmission between the first SDWAN and the secondSDWAN over the paths in the segment routing domain.

In certain embodiments, the performance measurements of the links in theWAN underlay can include one or a combination of congestion, latency, anumber of packet drops, and an amount of jitter in the links in the WANunderlay.

In various embodiments, the performance measurements can be collected asstreaming telemetry data from nodes forming the links in the WANunderlay.

In certain embodiments, paths in the segment routing domain between thefirst SDWAN and the second SDWAN can be identified. The paths in thesegment routing domain can be associated with specific traffic classesof data capable of being transmitted between the first SDWAN and thesecond SDWAN. In turn, transmission of data between the first SDWAN andthe second SDWAN over a specific path in the segment routing domain canbe controlled based on a traffic class of the data and associations ofthe paths with specific traffic classes of data. Two nodes in anidentified path in the segment routing domain between the first SDWANand the second SDWAN can be configured as path computation element nodesin the WAN underlay by a segment routing controller.

In various embodiments, performance healths of the paths in the segmentrouting domain between the first SDWAN and the second SDWAN can beascertained. A new path in the segment routing domain can be identifiedbased on the performance healths of the paths in the segment routingdomain. The new path can be associated with a specific traffic class ofthe data capable of being transmitted between the first SDWAN and thesecond SDWAN. Transmission of data of the specific traffic class throughthe new path can be controlled based on an association of the new pathto the specific traffic class. The new path can be associated with thespecific traffic class to replace a path previously associated with thespecific traffic class. Further, the new path can be identified based onquality of service requirements for transmitting the data of thespecific traffic class between the first SDWAN and the second SDWAN.

In certain embodiments quality of service requirements for transmittingdata of a specific traffic class between the first SDWAN and the secondSDWAN can be ascertained. An appropriate path in the segment routingdomain between the first SDWAN and the SDWAN can be identified based onthe quality of service requirements. In turn, transmission of data ofthe specific traffic class over the appropriate path in the segmentrouting domain between the first SDWAN and the second SDWAN can becontrolled through segment routing.

In various embodiments, the segment routing domain can utilize mediaaccess control security (MACsec) encryption to transmit data between thefirst SDWAN and the second SDWAN.

A system can include one or more processors and at least onecomputer-readable storage medium storing instructions which, whenexecuted by the one or more processors, cause the one or more processorsto identify a first SWAN and a second SDWAN of a SDWAN fabric. Theinstructions can also cause the one or more processors to form a segmentrouting domain through the SDWAN fabric that interconnects the firstSDWAN and the second SDWAN across WAN underlay of the SDWAN fabric bypre-building a plurality of selectable paths through the WAN underlaybetween the first SDWAN and the second SDWAN. Further, the instructionscan cause the one or more processors to control data transmissionbetween the first SDWAN and the second SDWAN by performing segmentrouting through the segment routing domain formed between the firstSDWAN and the second SDWAN.

A non-transitory computer-readable storage medium having stored thereininstructions which, when executed by a processor, cause the processor toidentify a first SDWAN and a second SDWAN of a SDWAN fabric. Theinstructions can also cause the processor to form a segment routingdomain through the SDWAN fabric that interconnects the first SDWAN andthe second SDWAN across a WAN underlay of the SDWAN fabric. Further, theinstructions can cause the processor to control data transmissionbetween the first SDWAN and the second SDWAN by performing segmentrouting through one or more changeable paths in the segment routingdomain formed between the first SDWAN and the second SDWAN.

EXAMPLE EMBODIMENTS

FIG. 1 illustrates an example of a network architecture 100 forimplementing aspects of the present technology. An example of animplementation of the network architecture 100 is the Cisco® SDWANarchitecture. However, one of ordinary skill in the art will understandthat, for the network architecture 100 and any other system discussed inthe present disclosure, there can be additional or fewer component insimilar or alternative configurations. The illustrations and examplesprovided in the present disclosure are for conciseness and clarity.Other embodiments may include different numbers and/or types of elementsbut one of ordinary skill the art will appreciate that such variationsdo not depart from the scope of the present disclosure.

In this example, the network architecture 100 can comprise anorchestration plane 102, a management plane 120, a control plane 130,and a data plane 140. The orchestration plane can 102 assist in theautomatic on-boarding of edge network devices 142 (e.g., switches,routers, etc.) in an overlay network. The orchestration plane 102 caninclude one or more physical or virtual network orchestrator appliances104. The network orchestrator appliance(s) 104 can perform the initialauthentication of the edge network devices 142 and orchestrateconnectivity between devices of the control plane 130 and the data plane140. In some embodiments, the network orchestrator appliance(s) 104 canalso enable communication of devices located behind Network AddressTranslation (NAT). In some embodiments, physical or virtual Cisco®SD-WAN vBond appliances can operate as the network orchestratorappliance(s) 104.

The management plane 120 can be responsible for central configurationand monitoring of a network. The management plane 120 can include one ormore physical or virtual network management appliances 122. In someembodiments, the network management appliance(s) 122 can providecentralized management of the network via a graphical user interface toenable a user to monitor, configure, and maintain the edge networkdevices 142 and links (e.g., Internet transport network 160, MPLSnetwork 162, 4G/LTE network 164) in an underlay and overlay network. Thenetwork management appliance(s) 122 can support multi-tenancy and enablecentralized management of logically isolated networks associated withdifferent entities (e.g., enterprises, divisions within enterprises,groups within divisions, etc.). Alternatively or in addition, thenetwork management appliance(s) 122 can be a dedicated networkmanagement system for a single entity. In some embodiments, physical orvirtual Cisco® SD-WAN vManage appliances can operate as the networkmanagement appliance(s) 122.

The control plane 130 can build and maintain a network topology and makedecisions on where traffic flows. The control plane 130 can include oneor more physical or virtual network controller appliance(s) 132. Thenetwork controller appliance(s) 132 can establish secure connections toeach network device 142 and distribute route and policy information viaa control plane protocol (e.g., Overlay Management Protocol (OMP)(discussed in further detail below), Open Shortest Path First (OSPF),Intermediate System to Intermediate System (IS-IS), Border GatewayProtocol (BGP), Protocol-Independent Multicast (PIM), Internet GroupManagement Protocol (IGMP), Internet Control Message Protocol (ICMP),Address Resolution Protocol (ARP), Bidirectional Forwarding Detection(BFD), Link Aggregation Control Protocol (LACP), etc.). In someembodiments, the network controller appliance(s) 132 can operate asroute reflectors. The network controller appliance(s) 132 can alsoorchestrate secure connectivity in the data plane 140 between and amongthe edge network devices 142. For example, in some embodiments, thenetwork controller appliance(s) 132 can distribute crypto keyinformation among the network device(s) 142. This can allow the networkto support a secure network protocol or application (e.g., InternetProtocol Security (IPSec), Transport Layer Security (TLS), Secure Shell(SSH), etc.) without Internet Key Exchange (IKE) and enable scalabilityof the network. In some embodiments, physical or virtual Cisco® SD-WANvSmart controllers can operate as the network controller appliance(s)132.

The data plane 140 can be responsible for forwarding packets based ondecisions from the control plane 130. The data plane 140 can include theedge network devices 142, which can be physical or virtual networkdevices. The edge network devices 142 can operate at the edges variousnetwork environments of an organization, such as in one or more datacenters or colocation centers 150, campus networks 152, branch officenetworks 154, home office networks 154, and so forth, or in the cloud(e.g., Infrastructure as a Service (IaaS), Platform as a Service (PaaS),SaaS, and other cloud service provider networks). The edge networkdevices 142 can provide secure data plane connectivity among sites overone or more WAN transports, such as via one or more Internet transportnetworks 160 (e.g., Digital Subscriber Line (DSL), cable, etc.), MPLSnetworks 162 (or other private packet-switched network (e.g., MetroEthernet, Frame Relay, Asynchronous Transfer Mode (ATM), etc.), mobilenetworks 164 (e.g., 3G, 4G/LTE, 5G, etc.), or other WAN technology(e.g., Synchronous Optical Networking (SONET), Synchronous DigitalHierarchy (SDH), Dense Wavelength Division Multiplexing (DWDM), or otherfiber-optic technology; leased lines (e.g., T1/E1, T3/E3, etc.); PublicSwitched Telephone Network (PSTN), Integrated Services Digital Network(ISDN), or other private circuit-switched network; small apertureterminal (VSAT) or other satellite network; etc.). The edge networkdevices 142 can be responsible for traffic forwarding, security,encryption, quality of service (QoS), and routing (e.g., BGP, OSPF,etc.), among other tasks. In some embodiments, physical or virtualCisco® SD-WAN vEdge routers can operate as the edge network devices 142.

FIG. 2 illustrates an example of a network topology 200 for showingvarious aspects of the network architecture 100. The network topology200 can include a management network 202, a pair of network sites 204Aand 204B (collectively, 204) (e.g., the data center(s) 150, the campusnetwork(s) 152, the branch office network(s) 154, the home officenetwork(s) 156, cloud service provider network(s), etc.), and a pair ofInternet transport networks 160A and 160B (collectively, 160). Themanagement network 202 can include one or more network orchestratorappliances 104, one or more network management appliance 122, and one ormore network controller appliances 132. Although the management network202 is shown as a single network in this example, one of ordinary skillin the art will understand that each element of the management network202 can be distributed across any number of networks and/or beco-located with the sites 204. In this example, each element of themanagement network 202 can be reached through either transport network160A or 160B.

Each site can include one or more endpoints 206 connected to one or moresite network devices 208. The endpoints 206 can include general purposecomputing devices (e.g., servers, workstations, desktop computers,etc.), mobile computing devices (e.g., laptops, tablets, mobile phones,etc.), wearable devices (e.g., watches, glasses or other head-mounteddisplays (HMDs), ear devices, etc.), and so forth. The endpoints 206 canalso include Internet of Things (IoT) devices or equipment, such asagricultural equipment (e.g., livestock tracking and management systems,watering devices, unmanned aerial vehicles (UAVs), etc.); connected carsand other vehicles; smart home sensors and devices (e.g., alarm systems,security cameras, lighting, appliances, media players, HVAC equipment,utility meters, windows, automatic doors, door bells, locks, etc.);office equipment (e.g., desktop phones, copiers, fax machines, etc.);healthcare devices (e.g., pacemakers, biometric sensors, medicalequipment, etc.); industrial equipment (e.g., robots, factory machinery,construction equipment, industrial sensors, etc.); retail equipment(e.g., vending machines, point of sale (POS) devices, Radio FrequencyIdentification (RFID) tags, etc.); smart city devices (e.g., streetlamps, parking meters, waste management sensors, etc.); transportationand logistical equipment (e.g., turnstiles, rental car trackers,navigational devices, inventory monitors, etc.); and so forth.

The site network devices 208 can include physical or virtual switches,routers, and other network devices. Although the site 204A is shownincluding a pair of site network devices and the site 204B is shownincluding a single site network device in this example, the site networkdevices 208 can comprise any number of network devices in any networktopology, including multi-tier (e.g., core, distribution, and accesstiers), spine-and-leaf, mesh, tree, bus, hub and spoke, and so forth.For example, in some embodiments, one or more data center networks mayimplement the Cisco® Application Centric Infrastructure (ACI)architecture and/or one or more campus networks may implement the Cisco®Software Defined Access (SD-Access or SDA) architecture. The sitenetwork devices 208 can connect the endpoints 206 to one or more edgenetwork devices 142, and the edge network devices 142 can be used todirectly connect to the transport networks 160.

In some embodiments, “color” can be used to identify an individual WANtransport network, and different WAN transport networks may be assigneddifferent colors (e.g., mpls, private1, biz-internet, metro-ethernet,lte, etc.). In this example, the network topology 200 can utilize acolor called “biz-internet” for the Internet transport network 160A anda color called “public-internet” for the Internet transport network160B.

In some embodiments, each edge network device 208 can form a DatagramTransport Layer Security (DTLS) or TLS control connection to the networkcontroller appliance(s) 132 and connect to any network control appliance132 over each transport network 160. In some embodiments, the edgenetwork devices 142 can also securely connect to edge network devices inother sites via IPSec tunnels. In some embodiments, the BFD protocol maybe used within each of these tunnels to detect loss, latency, jitter,and path failures.

On the edge network devices 142, color can be used help to identify ordistinguish an individual WAN transport tunnel (e.g., no same color maybe used twice on a single edge network device). Colors by themselves canalso have significance. For example, the colors metro-ethernet, mpls,and private1, private2, private3, private4, private5, and private6 maybe considered private colors, which can be used for private networks orin places where there is no NAT addressing of the transport IP endpoints(e.g., because there may be no NAT between two endpoints of the samecolor). When the edge network devices 142 use a private color, they mayattempt to build IPSec tunnels to other edge network devices usingnative, private, underlay IP addresses. The public colors can include 3g, biz, internet, blue, bronze, custom1, custom2, custom3, default,gold, green, lte, public-internet, red, and silver. The public colorsmay be used by the edge network devices 142 to build tunnels to post-NATIP addresses (if there is NAT involved). If the edge network devices 142use private colors and need NAT to communicate to other private colors,the carrier setting in the configuration can dictate whether the edgenetwork devices 142 use private or public IP addresses. Using thissetting, two private colors can establish a session when one or both areusing NAT.

FIG. 3 illustrates an example of a diagram 300 showing the operation ofOMP, which may be used in some embodiments to manage an overlay of anetwork (e.g., the network architecture 100). In this example, OMPmessages 302A and 302B (collectively, 302) may be transmitted back andforth between the network controller appliance 132 and the edge networkdevices 142A and 142B, respectively, where control plane information,such as route prefixes, next-hop routes, crypto keys, policyinformation, and so forth, can be exchanged over respective secure DTLSor TLS connections 304A and 304B. The network controller appliance 132can operate similarly to a route reflector. For example, the networkcontroller appliance 132 can receive routes from the edge networkdevices 142, process and apply any policies to them, and advertiseroutes to other edge network devices 142 in the overlay. If there is nopolicy defined, the edge network devices 142 may behave in a mannersimilar to a full mesh topology, where each edge network device 142 canconnect directly to another edge network device 142 at another site andreceive full routing information from each site.

OMP can advertise three types of routes:

-   -   OMP routes, which can correspond to prefixes that are learned        from the local site, or service side, of the edge network device        142. The prefixes can be originated as static or connected        routes, or from within, for example, the OSPF or BGP protocols,        and redistributed into OMP so they can be carried across the        overlay. OMP routes can advertise attributes such as transport        location (TLOC) information (which can similar to a BGP next-hop        IP address) and other attributes such as origin, originator,        preference, site identifier, tag, and virtual private network        (VPN). An OMP route may be installed in the forwarding table if        the TLOC to which it points is active.    -   TLOC routes, which can correspond to logical tunnel termination        points on the edge network devices 142 that connect into the        transport networks 160. In some embodiments, a TLOC route can be        uniquely identified and represented by a three-tuple, including        an IP address, link color, and encapsulation (e.g., Generic        Routing Encapsulation (GRE), IPSec, etc.). In addition to system        IP address, color, and encapsulation, TLOC routes can also carry        attributes such as TLOC private and public IP addresses,        carrier, preference, site identifier, tag, and weight. In some        embodiments, a TLOC may be in an active state on a particular        edge network device 142 when an active BFD session is associated        with that TLOC.    -   Service routes, which can represent services (e.g., firewall,        distributed denial of service (DDoS) mitigator, load balancer,        intrusion prevent system (IPS), intrusion detection systems        (IDS), WAN optimizer, etc.) that may be connected to the local        sites of the edge network devices 142 and accessible to other        sites for use with service insertion. In addition, these routes        can also include VPNs; the VPN labels can be sent in an update        type to tell the network controller appliance 132 what VPNs are        serviced at a remote site.

In the example of FIG. 3, OMP is shown running over the DTLS/TLS tunnels304 established between the edge network devices 142 and the networkcontroller appliance 132. In addition, the diagram 300 shows an IPSectunnel 306A established between TLOC 308A and 308C over the WANtransport network 160A and an IPSec tunnel 306B established between TLOC308B and TLOC 308D over the WAN transport network 160B. Once the IPSectunnels 306A and 306B are established, BFD can be enabled across each ofthem.

FIG. 4 illustrates an example of a diagram 400 showing the operation ofVPNs, which may be used in some embodiments to provide segmentation fora network (e.g., the network architecture 100). VPNs can be isolatedfrom one another and can have their own forwarding tables. An interfaceor sub-interface can be explicitly configured under a single VPN and maynot be part of more than one VPN. Labels may be used in OMP routeattributes and in the packet encapsulation, which can identify the VPNto which a packet belongs. The VPN number can be a four-byte integerwith a value from 0 to 65530. In some embodiments, the networkorchestrator appliance(s) 104, network management appliance(s) 122,network controller appliance(s) 132, and/or edge network device(s) 142can each include a transport VPN 402 (e.g., VPN number 0) and amanagement VPN 404 (e.g., VPN number 512). The transport VPN 402 caninclude one or more physical or virtual network interfaces (e.g.,network interfaces 410A and 410B) that respectively connect to WANtransport networks (e.g., the MPLS network 162 and the Internettransport network 160). Secure DTLS/TLS connections to the networkcontroller appliance(s) 132 or between the network controllerappliance(s) 132 and the network orchestrator appliance(s) 104 can beinitiated from the transport VPN 402. In addition, static or defaultroutes or a dynamic routing protocol can be configured inside thetransport VPN 402 to get appropriate next-hop information so that thecontrol plane 130 may be established and IPSec tunnels 306 (not shown)can connect to remote sites.

The management VPN 404 can carry out-of-band management traffic to andfrom the network orchestrator appliance(s) 104, network managementappliance(s) 122, network controller appliance(s) 132, and/or edgenetwork device(s) 142 over a network interface 410C. In someembodiments, the management VPN 404 may not be carried across theoverlay network.

In addition to the transport VPN 402 and the management VPN 404, thenetwork orchestrator appliance(s) 104, network management appliance(s)122, network controller appliance(s) 132, or edge network device(s) 142can also include one or more service-side VPNs 406. The service-side VPN406 can include one or more physical or virtual network interfaces(e.g., network interfaces 410D and 410E) that connect to one or morelocal-site networks 412 and carry user data traffic. The service-sideVPN(s) 406 can be enabled for features such as OSPF or BGP, VirtualRouter Redundancy Protocol (VRRP), QoS, traffic shaping, policing, andso forth. In some embodiments, user traffic can be directed over IPSectunnels to other sites by redistributing OMP routes received from thenetwork controller appliance(s) 132 at the site 412 into theservice-side VPN routing protocol. In turn, routes from the local site412 can be advertised to other sites by advertising the service VPNroutes into the OMP routing protocol, which can be sent to the networkcontroller appliance(s) 132 and redistributed to other edge networkdevices 142 in the network. Although the network interfaces 410A-E(collectively, 410) are shown to be physical interfaces in this example,one of ordinary skill in the art will appreciate that the interfaces 410in the transport and service VPNs can also be sub-interfaces instead.

FIG. 5A illustrates a diagram of an example Network Environment 500,such as a data center. In some cases, the Network Environment 500 caninclude a data center, which can support and/or host a cloudenvironment. The Network Environment 500 can include a Fabric 520 whichcan represent the physical layer or infrastructure (e.g., underlay) ofthe Network Environment 500. Fabric 520 can include Spines 502 (e.g.,spine routers or switches) and Leafs 504 (e.g., leaf routers orswitches) which can be interconnected for routing or switching trafficin the Fabric 520. Spines 502 can interconnect Leafs 504 in the Fabric520, and Leafs 504 can connect the Fabric 520 to an overlay or logicalportion of the Network Environment 500, which can include applicationservices, servers, virtual machines, containers, endpoints, etc. Thus,network connectivity in the Fabric 520 can flow from Spines 502 to Leafs504, and vice versa. The interconnections between Leafs 504 and Spines502 can be redundant (e.g., multiple interconnections) to avoid afailure in routing. In some embodiments, Leafs 504 and Spines 502 can befully connected, such that any given Leaf is connected to each of theSpines 502, and any given Spine is connected to each of the Leafs 504.Leafs 504 can be, for example, top-of-rack (“ToR”) switches, aggregationswitches, gateways, ingress and/or egress switches, provider edgedevices, and/or any other type of routing or switching device.

Leafs 504 can be responsible for routing and/or bridging tenant orcustomer packets and applying network policies or rules. Networkpolicies and rules can be driven by one or more Controllers 516, and/orimplemented or enforced by one or more devices, such as Leafs 504. Leafs504 can connect other elements to the Fabric 520. For example, Leafs 504can connect Servers 506, Hypervisors 508, Virtual Machines (VMs) 510,Applications 512, Network Device 514, etc., with Fabric 520. Suchelements can reside in one or more logical or virtual layers ornetworks, such as an overlay network. In some cases, Leafs 504 canencapsulate and decapsulate packets to and from such elements (e.g.,Servers 506) in order to enable communications throughout NetworkEnvironment 500 and Fabric 520. Leafs 504 can also provide any otherdevices, services, tenants, or workloads with access to Fabric 520. Insome cases, Servers 506 connected to Leafs 504 can similarly encapsulateand decapsulate packets to and from Leafs 504. For example, Servers 506can include one or more virtual switches or routers or tunnel endpointsfor tunneling packets between an overlay or logical layer hosted by, orconnected to, Servers 506 and an underlay layer represented by Fabric520 and accessed via Leafs 504.

Applications 512 can include software applications, services,containers, appliances, functions, service chains, etc. For example,Applications 512 can include a firewall, a database, a CDN server, anIDS/IPS, a deep packet inspection service, a message router, a virtualswitch, etc. An application from Applications 512 can be distributed,chained, or hosted by multiple endpoints (e.g., Servers 506, VMs 510,etc.), or may run or execute entirely from a single endpoint.

VMs 510 can be virtual machines hosted by Hypervisors 508 or virtualmachine managers running on Servers 506. VMs 510 can include workloadsrunning on a guest operating system on a respective server. Hypervisors508 can provide a layer of software, firmware, and/or hardware thatcreates, manages, and/or runs the VMs 510. Hypervisors 508 can allow VMs510 to share hardware resources on Servers 506, and the hardwareresources on Servers 506 to appear as multiple, separate hardwareplatforms. Moreover, Hypervisors 508 on Servers 506 can host one or moreVMs 510.

In some cases, VMs 510 can be migrated to other Servers 506. Servers 506can similarly be migrated to other physical locations in NetworkEnvironment 500. For example, a server connected to a specific leaf canbe changed to connect to a different or additional leaf. Suchconfiguration or deployment changes can involve modifications tosettings, configurations and policies that are applied to the resourcesbeing migrated as well as other network components.

In some cases, one or more Servers 506, Hypervisors 508, and/or VMs 510can represent or reside in a tenant or customer space. Tenant space caninclude workloads, services, applications, devices, networks, and/orresources that are associated with one or more clients or subscribers.Accordingly, traffic in Network Environment 500 can be routed based onspecific tenant policies, spaces, agreements, configurations, etc.Moreover, addressing can vary between one or more tenants. In someconfigurations, tenant spaces can be divided into logical segmentsand/or networks and separated from logical segments and/or networksassociated with other tenants. Addressing, policy, security andconfiguration information between tenants can be managed by Controllers516, Servers 506, Leafs 504, etc.

Configurations in Network Environment 500 can be implemented at alogical level, a hardware level (e.g., physical), and/or both. Forexample, configurations can be implemented at a logical and/or hardwarelevel based on endpoint or resource attributes, such as endpoint typesand/or application groups or profiles, through a software-definednetworking (SDN) framework (e.g., ACI or VMWARE NSX). To illustrate, oneor more administrators can define configurations at a logical level(e.g., application or software level) through Controllers 516, which canimplement or propagate such configurations through Network Environment500. In some examples, Controllers 516 can be Application PolicyInfrastructure Controllers (APICs) in an ACI framework. In otherexamples, Controllers 516 can be one or more management components forassociated with other SDN solutions, such as NSX Managers.

Such configurations can define rules, policies, priorities, protocols,attributes, objects, etc., for routing and/or classifying traffic inNetwork Environment 500. For example, such configurations can defineattributes and objects for classifying and processing traffic based onEndpoint Groups, Security Groups (SGs), VM types, bridge domains (BDs),virtual routing and forwarding instances (VRFs), tenants, priorities,firewall rules, etc. Other example network objects and configurationsare further described below. Traffic policies and rules can be enforcedbased on tags, attributes, or other characteristics of the traffic, suchas protocols associated with the traffic, EPGs associated with thetraffic, SGs associated with the traffic, network address informationassociated with the traffic, etc. Such policies and rules can beenforced by one or more elements in Network Environment 500, such asLeafs 504, Servers 506, Hypervisors 508, Controllers 516, etc. Aspreviously explained, Network Environment 500 can be configuredaccording to one or more particular SDN solutions, such as CISCO ACI orVMWARE NSX. These example SDN solutions are briefly described below.

ACI can provide an application-centric or policy-based solution throughscalable distributed enforcement. ACI supports integration of physicaland virtual environments under a declarative configuration model fornetworks, servers, services, security, requirements, etc. For example,the ACI framework implements EPGs, which can include a collection ofendpoints or applications that share common configuration requirements,such as security, QoS, services, etc. Endpoints can be virtual/logicalor physical devices, such as VMs, containers, hosts, or physical serversthat are connected to Network Environment 500. Endpoints can have one ormore attributes such as a VM name, guest OS name, a security tag,application profile, etc. Application configurations can be appliedbetween EPGs, instead of endpoints directly, in the form of contracts.Leafs 504 can classify incoming traffic into different EPGs. Theclassification can be based on, for example, a network segmentidentifier such as a VLAN ID, VXLAN Network Identifier (VNID), NVGREVirtual Subnet Identifier (VSID), MAC address, IP address, etc.

In some cases, classification in the ACI infrastructure can beimplemented by ACI virtual edge (AVE), which can run on a host, such asa server, e.g. a vSwitch running on a server. For example, the AVE canclassify traffic based on specified attributes, and tag packets ofdifferent attribute EPGs with different identifiers, such as networksegment identifiers (e.g., VLAN ID). Finally, Leafs 504 can tie packetswith their attribute EPGs based on their identifiers and enforcepolicies, which can be implemented and/or managed by one or moreControllers 516. Leaf 504 can classify to which EPG the traffic from ahost belongs and enforce policies accordingly.

Another example SDN solution is based on VMWARE NSX. With VMWARE NSX,hosts can run a distributed firewall (DFW) which can classify andprocess traffic. Consider a case where three types of VMs, namely,application, database and web VMs, are put into a single layer-2 networksegment. Traffic protection can be provided within the network segmentbased on the VM type. For example, HTTP traffic can be allowed among webVMs, and disallowed between a web VM and an application or database VM.To classify traffic and implement policies, VMWARE NSX can implementsecurity groups, which can be used to group the specific VMs (e.g., webVMs, application VMs, database VMs). DFW rules can be configured toimplement policies for the specific security groups. To illustrate, inthe context of the previous example, DFW rules can be configured toblock HTTP traffic between web, application, and database securitygroups.

Returning now to FIG. 5A, Network Environment 500 can deploy differenthosts via Leafs 504, Servers 506, Hypervisors 508, VMs 510, Applications512, and Controllers 516, such as VMWARE ESXi hosts, WINDOWS HYPER-Vhosts, bare metal physical hosts, etc. Network Environment 500 mayinteroperate with a variety of Hypervisors 508, Servers 506 (e.g.,physical and/or virtual servers), SDN orchestration platforms, etc.Network Environment 200 may implement a declarative model to allow itsintegration with application design and holistic network policy.

Controllers 516 can provide centralized access to fabric information,application configuration, resource configuration, application-levelconfiguration modeling for a SDN infrastructure, integration withmanagement systems or servers, etc. Controllers 516 can form a controlplane that interfaces with an application plane via northbound APIs anda data plane via southbound APIs.

As previously noted, Controllers 516 can define and manageapplication-level model(s) for configurations in Network Environment500. In some cases, application or device configurations can also bemanaged and/or defined by other components in the network. For example,a hypervisor or virtual appliance, such as a VM or container, can run aserver or management tool to manage software and services in NetworkEnvironment 500, including configurations and settings for virtualappliances.

As illustrated above, Network Environment 500 can include one or moredifferent types of SDN solutions, hosts, etc. For the sake of clarityand explanation purposes, various examples in the disclosure will bedescribed with reference to an ACI framework, and Controllers 516 may beinterchangeably referenced as controllers, APICs, or APIC controllers.However, it should be noted that the technologies and concepts hereinare not limited to ACI solutions and may be implemented in otherarchitectures and scenarios, including other SDN solutions as well asother types of networks which may not deploy an SDN solution.

Further, as referenced herein, the term “hosts” can refer to Servers 506(e.g., physical or logical), Hypervisors 508, VMs 510, containers (e.g.,Applications 512), etc., and can run or include any type of server orapplication solution. Non-limiting examples of “hosts” can includevirtual switches or routers, such as distributed virtual switches (DVS),AVE nodes, vector packet processing (VPP) switches; VCENTER and NSXMANAGERS; bare metal physical hosts; HYPER-V hosts; VMs; DOCKERContainers; etc.

FIG. 5B illustrates another example of Network Environment 500. In thisexample, Network Environment 500 includes Endpoints 522 connected toLeafs 504 in Fabric 520. Endpoints 522 can be physical and/or logical orvirtual entities, such as servers, clients, VMs, hypervisors, softwarecontainers, applications, resources, network devices, workloads, etc.For example, an Endpoint 522 can be an object that represents a physicaldevice (e.g., server, client, switch, etc.), an application (e.g., webapplication, database application, etc.), a logical or virtual resource(e.g., a virtual switch, a virtual service appliance, a virtualizednetwork function (VNF), a VM, a service chain, etc.), a containerrunning a software resource (e.g., an application, an appliance, a VNF,a service chain, etc.), storage, a workload or workload engine, etc.Endpoints 522 can have an address (e.g., an identity), a location (e.g.,host, network segment, VRF instance, domain, etc.), one or moreattributes (e.g., name, type, version, patch level, OS name, OS type,etc.), a tag (e.g., security tag), a profile, etc.

Endpoints 522 can be associated with respective Logical Groups 518.Logical Groups 518 can be logical entities containing endpoints(physical and/or logical or virtual) grouped together according to oneor more attributes, such as endpoint type (e.g., VM type, workload type,application type, etc.), one or more requirements (e.g., policyrequirements, security requirements, QoS requirements, customerrequirements, resource requirements, etc.), a resource name (e.g., VMname, application name, etc.), a profile, platform or operating system(OS) characteristics (e.g., OS type or name including guest and/or hostOS, etc.), an associated network or tenant, one or more policies, a tag,etc. For example, a logical group can be an object representing acollection of endpoints grouped together. To illustrate, Logical Group 1can contain client endpoints, Logical Group 2 can contain web serverendpoints, Logical Group 3 can contain application server endpoints,Logical Group N can contain database server endpoints, etc. In someexamples, Logical Groups 518 are EPGs in an ACI environment and/or otherlogical groups (e.g., SGs) in another SDN environment.

Traffic to and/or from Endpoints 522 can be classified, processed,managed, etc., based Logical Groups 518. For example, Logical Groups 518can be used to classify traffic to or from Endpoints 522, apply policiesto traffic to or from Endpoints 522, define relationships betweenEndpoints 522, define roles of Endpoints 522 (e.g., whether an endpointconsumes or provides a service, etc.), apply rules to traffic to or fromEndpoints 522, apply filters or access control lists (ACLs) to trafficto or from Endpoints 522, define communication paths for traffic to orfrom Endpoints 522, enforce requirements associated with Endpoints 522,implement security and other configurations associated with Endpoints522, etc.

In an ACI environment, Logical Groups 518 can be EPGs used to definecontracts in the ACI. Contracts can include rules specifying what andhow communications between EPGs take place. For example, a contract candefine what provides a service, what consumes a service, and what policyobjects are related to that consumption relationship. A contract caninclude a policy that defines the communication path and all relatedelements of a communication or relationship between EPs or EPGs. Forexample, a Web EPG can provide a service that a Client EPG consumes, andthat consumption can be subject to a filter (ACL) and a service graphthat includes one or more services, such as firewall inspection servicesand server load balancing.

As discussed previously, the enterprise network landscape iscontinuously evolving. There is a greater demand for mobile and IoTdevice traffic, SaaS applications, and cloud adoption. In addition,security needs are increasing and certain applications can requireprioritization and optimization for proper operation. As this complexitygrows, there is a push to reduce costs and operating expenses whileproviding for high availability and scale.

Conventional WAN architectures are facing major challenges under thisevolving landscape. Conventional WAN architectures typically consist ofmultiple MPLS transports, or MPLS paired with Internet or LTE links usedin an active/backup fashion, most often with Internet or SaaS trafficbeing backhauled to a central data center or regional hub for Internetaccess. Issues with these architectures can include insufficientbandwidth, high bandwidth costs, application downtime, poor SaaSperformance, complex operations, complex workflows for cloudconnectivity, long deployment times and policy changes, limitedapplication visibility, and difficulty in securing the network.

In recent years, software-defined enterprise network solutions have beendeveloped to address these challenges. Software-defined enterprisenetworking is part of a broader technology of SDN that includes bothSDWANs and SDLANs. SDN is a centralized approach to network managementwhich can abstract away the underlying network infrastructure from itsapplications. This de-coupling of data plane forwarding and controlplane can allow a network operator to centralize the intelligence of thenetwork and provide for more network automation, operationssimplification, and centralized provisioning, monitoring, andtroubleshooting. Software-defined enterprise networking can apply theseprinciples of SDN to the WAN and a LAN.

Currently SDWANs can be combined to form a single network, e.g. a verylarge network. For example, regional campus networks can form a verylarge network of one or more entities. Specifically, instead of buildingone large SDWAN, a hierarchy of SDWANs can be created to form a networkby building regional SD-WAN networks/clouds. Often these regional SDWANsare terminated at hub sites, Data Centers and/or colocation facilities.In forming a network through a plurality of SDWANs, facilitatingcommunication between the SDWANs, e.g. interconnecting the SDWANs, iscritical to ensuring that the network functions properly. However,interconnecting separate SDWANs is difficult to accomplish.Specifically, interconnecting separate SDWANs through a SDWAN fabricsupporting the SDWANs is difficult to properly implement. Theretherefore exist needs for systems and methods of interconnectingseparate SDWANs forming a larger network. More specifically, there existneeds for systems and methods of interconnecting separate SDWANs througha SDWAN fabric in which the SDWANs are formed.

The present includes systems, methods, and computer-readable media forsolving these problems/discrepancies by interconnecting SDWANs throughsegment routing. Specifically, a first SDWAN and a second SDWAN of aSDWAN fabric can be identified. A segment routing domain can be formedthrough the SDWAN fabric that interconnects the first SDWAN and thesecond SDWAN. Specifically, the segment routing domain can be formedacross a WAN underlay of the SDWAN fabric to interconnect the firstSDWAN and the second SDWAN. Data transmission between the first SDWANand the second SDWAN can be controlled by performing segment routingthrough the segment routing domain formed between the first SDWAN andthe second SDWAN.

The present systems, methods, and computer-readable media areadvantageous over current cross-domain enforcement techniques for anumber of reasons. Specifically, the benefits of an end-to-end SDWANsolution can be realized by interconnecting SDWANs through a segmentrouting domain built on a WAN core/underlay, effectively connecting theSDWANs to create an end-to-end SDWAN solution. Further, using a segmentrouting domain to interconnect SDWANs provides functionalities forbuilding paths through the WAN core using software instantiatedconstructs (e.g. from a controller). As follows, due to the mechanismsof headend decision making and path selection that are characteristic ofsegment routing, paths for specific traffic can be picked based oncharacteristics of the traffic to more efficiently control datatransmission between SDWANs. Further through the use of segment routing,paths can be changed reactively to more efficiently control datatransmission between SDWANs. This is advantageous over typical networkstructures used to connect SDWANs, which are not usually capable ofimplementing path changes in controlling data transmission betweenSDWANs unless the path changes are pre-built.

FIG. 6 shows an example network environment 600 of interconnectedSDWANs. The example network environment 600 includes a first SDWAN 602and a second SDWAN 604. The first and second SDWANs 602 and 604 can beapplicable SDWANs in a network environment. For example, the first andsecond SDWANs 602 and 604 can be formed as part of datacenters, campusnetworks, regional office networks, and other applicable cloudenvironments. Further, the first and second SDWANs 602 and 604 can beimplemented and managed using an applicable SDN architecture. Forexample, the first and second SDWANs 602 and 604 can be implementedusing the network architecture 100 shown in FIG. 1. The first and secondSDWANs 602 and 604 can be part of the same enterprise network. Forexample, the first and second SDWANs 602 and 604 can include a campusnetwork and a datacenter of an enterprise at different locations.

The first and second SDWANs 602 and 604 can be formed as part of anSDWAN fabric. Specifically, the first and second SDWANs 602 and 604 canbe formed by an applicable underlay network, such as the underlaynetwork 606 shown in FIG. 6, of an SDWAN fabric. While the first andsecond SDWANs 602 and 604 are shown as being implemented separate fromthe underlay network 606, this is done for illustrative purposes, andall or portions of the first and second SDWANs 602 and 604 can actuallybe formed over the underlay network 606. Accordingly, the underlaynetwork 606 can form part of the SDWAN fabric that includes the firstand second SDWANs 602 and 604.

The underlay network 606 can be formed by one or more applicablenetworks. Specifically, the underlay network 606 can be formed through abroadband network, a MPLS network, a cellular network, and/or a privateinterconnect network. Further, the underlay network 606 can be formedthrough one or more networks of one or more network service providers.Specifically, the underlay network 606 can be formed by a cellularnetwork and a broadband network of different network service providers.The underlay network 606 can serve as a WAN underlay that interconnectsthe first SDWAN 602 and the second SDWAN 604. Specifically, the firstSDWAN 602 and the second SDWAN 604 can exchange data with each otheracross the underlay network 606, thereby interconnecting the first SDWAN602 and the second SDWAN 604.

A segment routing domain 608 can be formed through the underlay network606 to interconnect the first SDWAN 602 and the second SDWAN 604. Inturn, data can be transmitted between the first SDWAN 602 and the secondSDWAN 604 through the segment routing domain 608 to interconnect thefirst SDWAN 602 and the second SDWAN 604. Specifically, datatransmission between the first SDWAN 602 and the second SDWAN 604 can becontrolled using segment routing by controlling the transmission of databetween the first SDWAN 602 and the second SDWAN 604 through the segmentrouting domain 608. By interconnecting the first SDWAN 602 and thesecond SDWAN 604 through the segment routing domain 608, the benefits ofan end-to-end SDWAN solution can be realized. Specifically, the firstSDWAN 602 and the second SDWAN 604 can communicate more efficiently andmore securely with each other than typical solutions for interconnectingSDWANs, e.g. IPSEC solutions.

As part of forming the segment routing domain 608 in the underlaynetwork 606, one or more paths can be identified and built in theunderlay network 606 for connecting the SDWANs 602 and 604. The pathscan be formed from a plurality of applicable network devices in theunderlay network 606 to connect the SDWANs 602 and 604 through theunderlay network 606. In turn and as will be discussed in greater detaillater, the paths can be selectable based on traffic type, to controltransmission of traffic between the SDWANs 602 and 604 through thesegment routing domain 608 using segment routing. Subsequently, thetraffic can be transmitted over the selected paths as part oftransmitting the traffic between the first and second SDWANs 602 and 604using segment routing. The segment routing domain 608 can transmittraffic between the first SDWAN 602 and the second SDWAN 604 using mediaaccess control security (MACsec) encryption.

The paths through the segment routing domain 608 can be built by anapplicable software controller, e.g. a segment routing controller/pathcomputation element controller associated with the segment routingdomain 608. In establishing the pre-defined paths through the segmentrouting domain 608, the segment routing controller can configure thesegment routing domain 608 to route traffic between the SDWANS 602 and604 according to the selected path. Specifically, the segment routingcontroller can signal a list of segment(s) of the path to a head-endrouter/provider edge router in the segment routing domain 608. The listof segment(s) of the path can be used to program, at the provider edgerouter, a single per-flow state corresponding to the path. In turn, theprovider edge router can insert the list of segments into packet headersfor transmitting traffic through the path using segment routing.Further, the segment routing controller can add a binding segment ID(“BSID”) for the path to the provider edge router in the segment routingdomain 608. The BSID can be uniquely associated with or otherwiseidentify a specific policy for traffic associated with the path. Inturn, the policy can be used, e.g. by the provider edge router, totransmit the traffic through the path using segment routing.

In various embodiments, the provider edge router can function as a routecomputation element node in the WAN underlay. Specifically, the provideredge router can function as a route computation element node byinserting segments into packet headers and using the BSID/policy tocontrol traffic through the path. The segment routing controller canprogram two nodes in the path to function as route computation elementnodes. For example, the segment routing controller can signal the listof segment(s) of the path and the BSID to two provider edge routers inthe WAN underlay.

The path can be pushed, e.g. the choice of the path can be pushed, to anapplicable controller, e.g. an SDWAN controller for either or both thefirst SDWAN 602 and the second SDWAN 604. Specifically, the choice ofthe path can be pushed to the SDWAN controller after or in conjunctionwith the segment routing controller configuring the path according tothe previously described techniques. More specifically, the segmentrouting controller can push the BSID and other applicable attributes ofthe path, e.g. a list of segments in the path, to the SDWAN controller.In turn, the SDWAN controller can configure the first SDWAN 602 and thesecond SDWAN 604 to transmit traffic through the segment routing domain608 using the path established by the segment routing controller.Specifically, the SDWAN controller can configure edge routers in thefirst SDWAN 602 and the second SDWAN 604 with policies to facilitatetransmission of traffic through the path established by the segmentrouting controller through the segment routing domain 608.

Paths through the segment routing domain 608 can be uniquely associatedwith specific classes of traffic. In turn, traffic of a specific trafficclass can be transmitted between the first SDWAN 602 and the secondSDWAN 604 through a specific path in the segment routing domain that isassociated with the specific traffic class. A traffic class can bedefined by one or more applicable characteristics of traffic.Specifically, a traffic class can be defined based on one or acombination of a user group, an application group, a VPN group, asource, and a destination associated with traffic. For example, atraffic class can include data associated with a specific applicationexecuted as part of providing network service access. In anotherexample, a traffic class can include data that is transmitted to andfrom a specific client in providing network service access.

In transmitting traffic through the segment routing domain 608 based onspecific classes of the traffic, policies can be implemented to controltraffic transmission based on traffic classes. A policy can be defined,at least in part, by an administrator. For example, an administrator candefine a specific traffic class to control through the segment routingdomain 608. A policy can also identify a specific path through thesegment routing domain 608, e.g. as established by the segment routingcontroller, to transmit a specific class of traffic over. An applicablecontroller can implement a policy for controlling traffic transmissionthrough the segment routing domain 608 based on traffic type. Forexample, the SDWAN controller can program a policy onto applicable edgerouters in the first and second SDWANs 602 and 604 for controllingtraffic transmission through the segment routing domain 608 based ontraffic class. In another example, the segment routing controller canprogram an appropriate provider edge router in the underlay network 606with a BSID corresponding to a policy for transmitting traffic throughthe segment routing domain 608 based on traffic class.

A path for transmitting specific traffic, e.g. a specific traffic class,through the segment routing domain 608 can be selected based onperformance characteristics/measurements of links in the underlaynetwork 606. In particular, a specific path can be identified and builtbased on performance measurements of links in the underlay network 606,as part of performing segment routing based on the performancemeasurements. Performance measurements can include applicable metricsrelated to transmission of data through the links in the underlaynetwork 606. For example, performance measurements can include one or anapplicable combination of congestion in the links, latency in the links,a number of packet drops in the links, and an amount of jitter in thelinks. Performance measurements can be identified from telemetry datareceived from nodes forming the links in the underlay network 606. Thetelemetry data can be streamed from the nodes to an applicablecontroller, e.g. either or both the SDWAN controller and the segmentrouting controller. In turn, the controller can identify the performancemeasurements of the links in the underlay network 606 from the streamingtelemetry data.

Further, a path for transmitting specific traffic, e.g. traffic of aspecific traffic class, through the segment routing domain 608 can beselected based on quality of service requirements associated withtransmitting the specific traffic. Quality of service requirements canbe specific to traffic based on a traffic class. In turn, the paths fortransmitting specific classes of traffic through the segment routingdomain 608 can be identified and established based on the quality ofservice requirements of the specific traffic classes. Quality of servicerequirements can include applicable performance requirements fortransmitting traffic between the first SDWAN 602 and the second SDWAN604. Specifically, quality of service requirements can include either orboth bandwidth and latency thresholds/requirements for transmittingdata, e.g. of a specific traffic class, between the first SDWAN 602 andthe second SDWAN 604. Latency requirements can include threshold one-waydata transmission times and threshold round-trip data delay times fortransmitting data between sources and destinations. Bandwidthrequirements can include threshold data transmission rates fortransmitting data between sources and destinations. For example, qualityof service requirements of an application can specify that trafficassociated with the application should be transmitted between the firstSDWAN 602 and the second SDWAN 604 at a specific average bit rate over aspecific period of time.

A path for transmitting specific traffic e.g. a traffic class, throughthe segment routing domain 608 can be selected based on both quality ofservice requirements associated with the traffic and performancemeasurements of links in the underlay network 606. In turn, the selectedpath can be established/built through the segment routing domain 608 fortransmitting traffic between the first SDWAN 602 and the second SDWAN604 through segment routing. Specifically, a path can be identified andbuilt with links that meet quality of service requirements of specifictraffic. For example, the segment routing controller can select linksthat form a path having a bandwidth that meets a bandwidth threshold fortransmitting a specific class of traffic between the first SDWAN 602 andthe second SDWAN 604.

In an example of selecting a path through the segment routing domain 608based on both quality of service requirements and performancemeasurements of links in the underlay network 606, an SDWAN capablerouter, e.g. a vEdge Router®, in the first SDWAN 602 can identify anapplication associated with a class of traffic and choose a specificpath through the underlay network 606 to the second SDWAN 604. The SDWANcapable router can select the specific path based on performancemeasurements gathered for the underlay network 606. The path selected bythe SDWAN capable router might be the shortest path through the underlaynetwork 606. However, the path can still fail to meet quality of servicerequirements for the application, e.g. the latency threshold orbandwidth tolerance for the application. The router can then send theidentified path to the SDWAN controller. The SDWAN controller can sendthe identified path as well as the quality of service requirements forthe application to the segment routing controller, e.g. as part of arequest for establishing a path for the application through the underlaynetwork 606 in the segment routing domain 608. Further, the SDWANcontroller can send the sources and destinations in the first SDWAN 602and the second SDWAN 604 of traffic associated with the application.

Further in the example, the segment routing controller can identify apath through the underlay network 606 that meets quality of servicerequirements for the application. Specifically, the segment routingcontroller can identify the next shortest path, when compared to thepath identified by the SDWAN capable router, that meets the quality ofservice requirements for the application. The segment routing controllercan identify a path through the underlay network 606 that meets thequality of service requirements for the application based on performancemeasurements of the links, e.g. as determined from telemetry datagathered by nodes forming the links, in the underlay network 606.

Then, the segment routing controller can establish the path in thesegment routing domain 608 using the previously described techniques forestablishing a path through the segment routing domain 608.Specifically, the segment routing controller can program a provider edgerouter in the path with a list of the link segments in the path.Further, the segment routing controller can program the provider edgerouter with a BSID for a policy to control a flow of the applicationtraffic through the path. As follows, the segment routing controller cansend an identification of the path, e.g. the link segments in the path,the BSID for the policy of the path, and/or an indication that the pathis established. The SDWAN controller can then configure the first SDWAN602 and/or the second SDWAN 604 to transmit traffic of the applicationthrough the path in the segment routing domain 608. For example, theSDWAN controller can configure edge nodes in the first SDWAN 602, e.g.the SDWAN capable router, to transmit data of the application to thepath in the segment routing domain 608, e.g. the provider edge node ofthe path in the segment routing domain 608.

The previously described techniques can be used to synchronizeoperations of both the SDWAN controller and the segment routingcontroller to transmit data through the segment routing domain 608 whilemeeting quality of service requirements. In turn, this can allow theSDWAN controller to take advantage of available low-latency paths in theunderlay network 606 for SDWAN applications.

A path through the segment routing domain 608 that is used to transmittraffic of a specific class between the first SDWAN 602 and the secondSDWAN 604 can be modified to effectively create a new path used totransmit the traffic. Alternatively, an entirely new path through thesegment routing domain 608 can be formed for transmitting the traffic ofthe specific class between the first and second SDWANs 602 and 604. Anew path, e.g. a modified path or entirely new path, can be identifiedand associated with the specific traffic class to replace the pathpreviously associated with the specific traffic class. As follows, thetraffic of the specific class can be transmitted over the new ormodified path based on the association of the new or modified path withthe specific traffic class of the traffic.

The new path can be established based on monitored performance healthsof the paths through the segment routing domain 608. Health of a pathcan be defined according to performance measurements of the path intransmitting data between the first and second SDWANs 602 and 604.Specifically, the SDWAN controller can monitor a health of the pathtransmitting traffic through the segment routing domain based onperformance measurements of links in the path, e.g. as indicated byreceived telemetry data. For example, health of a path can include ascore that is determined based on load levels in the path, latency inthe path, a number of packet drops in the path, and/or jitter in thepath. In turn, the SDWAN controller can facilitate establishment of thenew path based on the health of the path with respect to a thresholdhealth level, e.g. if the health of the path drops below the thresholdhealth level.

The new path can be established using the techniques described herein.Specifically, the segment routing controller can identify the new path,e.g. by modifying the path through the segment routing domain 608 oridentifying an entirely new path through the segment routing domain 608.Further, the segment routing controller can identify the new path basedon quality of service requirements of traffic associated with theprevious path. For example, the segment routing controller can identifya new path that meets the bandwidth requirements of an application. Thesegment routing controller can identify a new path based on a pathidentified by the SDWAN controller, similar to the technique discussedpreviously with respect to the SDWAN controller identifying a path andthe segment routing controller identifying the next shortest path thatmeets quality of service requirements.

FIG. 7 illustrates an example of a network device 700 (e.g., switch,router, network appliance, etc.). The network device 700 can include amaster central processing unit (CPU) 702, interfaces 704, and a bus 706(e.g., a PCI bus). When acting under the control of appropriate softwareor firmware, the CPU 702 can be responsible for executing packetmanagement, error detection, and/or routing functions. The CPU 702preferably accomplishes all these functions under the control ofsoftware including an operating system and any appropriate applicationssoftware. The CPU 702 may include one or more processors 708 such as aprocessor from the Motorola family of microprocessors or the MIPS familyof microprocessors. In an alternative embodiment, the processor 708 canbe specially designed hardware for controlling the operations of thenetwork device 700. In an embodiment, a memory 710 (such as non-volatileRAM and/or ROM) can also form part of the CPU 702. However, there aremany different ways in which memory could be coupled to the system.

The interfaces 704 can be provided as interface cards (sometimesreferred to as line cards). The interfaces 704 can control the sendingand receiving of data packets over the network and sometimes supportother peripherals used with the network device 700. Among the interfacesthat may be provided are Ethernet interfaces, frame relay interfaces,cable interfaces, DSL interfaces, token ring interfaces, and the like.In addition, various very high-speed interfaces may be provided such asa fast token ring interface, wireless interface, Ethernet interface,Gigabit Ethernet interface, Asynchronous Transfer Mode (ATM) interface,High-Speed Serial Interface (HSSI), Packet Over SONET (POS) interface,Fiber Distributed Data Interface (FDDI), and the like. The interfaces904 may include ports appropriate for communication with the appropriatemedia. In some cases, the interfaces 704 may also include an independentprocessor and, in some instances, volatile RAM. The independentprocessors may control communication intensive tasks such as packetswitching, media control, and management. By providing separateprocessors for the communication intensive tasks, the interfaces 704 mayallow the CPU 702 to efficiently perform routing computations, networkdiagnostics, security functions, and so forth.

Although the system shown in FIG. 7 is an example of a network device ofan embodiment, it is by no means the only network device architecture onwhich the subject technology can be implemented. For example, anarchitecture having a single processor that can handle communications aswell as routing computations and other network functions, can also beused. Further, other types of interfaces and media may also be used withthe network device 700.

Regardless of the network device's configuration, it may employ one ormore memories or memory modules (including the memory 710) configured tostore program instructions for general-purpose network operations andmechanisms for roaming, route optimization, and routing functionsdescribed herein. The program instructions may control the operation ofan operating system and/or one or more applications. The memory ormemories may also be configured to store tables such as mobilitybinding, registration, and association tables.

FIG. 8 illustrates an example of a bus computing system 800 wherein thecomponents of the system are in electrical communication with each otherusing a bus 805. The computing system 800 can include a processing unit(CPU or processor) 810 and a system bus 805 that may couple varioussystem components including the system memory 815, such as read onlymemory (ROM) 820 and random access memory (RAM) 825, to the processor810. The computing system 800 can include a cache 812 of high-speedmemory connected directly with, in close proximity to, or integrated aspart of the processor 810. The computing system 800 can copy data fromthe memory 815, ROM 820, RAM 825, and/or storage device 830 to the cache812 for quick access by the processor 810. In this way, the cache 812can provide a performance boost that avoids processor delays whilewaiting for data. These and other modules can control the processor 810to perform various actions. Other system memory 815 may be available foruse as well. The memory 815 can include multiple different types ofmemory with different performance characteristics. The processor 810 caninclude any general purpose processor and a hardware module or softwaremodule, such as module 1 832, module 2 834, and module 3 836 stored inthe storage device 830, configured to control the processor 810 as wellas a special-purpose processor where software instructions areincorporated into the actual processor design. The processor 810 mayessentially be a completely self-contained computing system, containingmultiple cores or processors, a bus, memory controller, cache, etc. Amulti-core processor may be symmetric or asymmetric.

To enable user interaction with the computing system 800, an inputdevice 845 can represent any number of input mechanisms, such as amicrophone for speech, a touch-protected screen for gesture or graphicalinput, keyboard, mouse, motion input, speech and so forth. An outputdevice 835 can also be one or more of a number of output mechanismsknown to those of skill in the art. In some instances, multimodalsystems can enable a user to provide multiple types of input tocommunicate with the computing system 800. The communications interface840 can govern and manage the user input and system output. There may beno restriction on operating on any particular hardware arrangement andtherefore the basic features here may easily be substituted for improvedhardware or firmware arrangements as they are developed.

The storage device 830 can be a non-volatile memory and can be a harddisk or other types of computer readable media which can store data thatare accessible by a computer, such as magnetic cassettes, flash memorycards, solid state memory devices, digital versatile disks, cartridges,random access memory, read only memory, and hybrids thereof.

As discussed above, the storage device 830 can include the softwaremodules 832, 834, 836 for controlling the processor 810. Other hardwareor software modules are contemplated. The storage device 830 can beconnected to the system bus 805. In some embodiments, a hardware modulethat performs a particular function can include a software componentstored in a computer-readable medium in connection with the necessaryhardware components, such as the processor 810, bus 805, output device835, and so forth, to carry out the function.

For clarity of explanation, in some instances the various embodimentsmay be presented as including individual functional blocks includingfunctional blocks comprising devices, device components, steps orroutines in a method embodied in software, or combinations of hardwareand software.

In some embodiments the computer-readable storage devices, media, andmemories can include a cable or wireless signal containing a bit streamand the like. However, when mentioned, non-transitory computer-readablestorage media expressly exclude media such as energy, carrier signals,electromagnetic waves, and signals per se.

Methods according to the above-described examples can be implementedusing computer-executable instructions that are stored or otherwiseavailable from computer readable media. Such instructions can comprise,for example, instructions and data which cause or otherwise configure ageneral purpose computer, special purpose computer, or special purposeprocessing device to perform a certain function or group of functions.Portions of computer resources used can be accessible over a network.The computer executable instructions may be, for example, binaries,intermediate format instructions such as assembly language, firmware, orsource code. Examples of computer-readable media that may be used tostore instructions, information used, and/or information created duringmethods according to described examples include magnetic or opticaldisks, flash memory, USB devices provided with non-volatile memory,networked storage devices, and so on.

Devices implementing methods according to these disclosures can comprisehardware, firmware and/or software, and can take any of a variety ofform factors. Some examples of such form factors include general purposecomputing devices such as servers, rack mount devices, desktopcomputers, laptop computers, and so on, or general purpose mobilecomputing devices, such as tablet computers, smart phones, personaldigital assistants, wearable devices, and so on. Functionality describedherein also can be embodied in peripherals or add-in cards. Suchfunctionality can also be implemented on a circuit board among differentchips or different processes executing in a single device, by way offurther example.

The instructions, media for conveying such instructions, computingresources for executing them, and other structures for supporting suchcomputing resources are means for providing the functions described inthese disclosures.

Although a variety of examples and other information was used to explainaspects within the scope of the appended claims, no limitation of theclaims should be implied based on particular features or arrangements insuch examples, as one of ordinary skill would be able to use theseexamples to derive a wide variety of implementations. Further andalthough some subject matter may have been described in languagespecific to examples of structural features and/or method steps, it isto be understood that the subject matter defined in the appended claimsis not necessarily limited to these described features or acts. Forexample, such functionality can be distributed differently or performedin components other than those identified herein. Rather, the describedfeatures and steps are disclosed as examples of components of systemsand methods within the scope of the appended claims.

What is claimed is:
 1. A method comprising: identifying a firstsoftware-defined wide area network (SDWAN) and a second SDWAN of a SDWANfabric; forming a segment routing domain through the SDWAN fabric thatinterconnects the first SDWAN and the second SDWAN across a wide areanetwork (WAN) underlay of the SDWAN fabric; and controlling datatransmission between the first SDWAN and the second SDWAN by performingsegment routing through the segment routing domain formed between thefirst SDWAN and the second SDWAN.
 2. The method of claim 1, whereinforming the segment routing domain through the SDWAN fabric furthercomprises pre-building a plurality of paths through the WAN underlaybetween the first SDWAN and the second SDWAN, wherein the plurality ofpaths are selectable to control data transmission between the firstSDWAN and the second SDWAN through the segment routing domain.
 3. Themethod of claim 2, wherein the plurality of paths are changeable tocontrol data transmission between the first SDWAN and the second SDWANthrough the segment routing domain.
 4. The method of claim 1, furthercomprising: gathering performance measurements of links in the WANunderlay forming paths in the segment routing domain; and performingsegment routing through the segment routing domain based on theperformance measurements of the links in the WAN underlay to controldata transmission between the first SDWAN and the second SDWAN over thepaths in the segment routing domain.
 5. The method of claim 4, whereinthe performance measurements of the links in the WAN underlay includeone or a combination of congestion, latency, a number of packet drops,and an amount of jitter in the links in the WAN underlay.
 6. The methodof claim 4, wherein the performance measurements are collected asstreaming telemetry data from nodes forming the links in the WANunderlay.
 7. The method of claim 1, further comprising: identifyingpaths in the segment routing domain between the first SDWAN and thesecond SDWAN; associating the paths in the segment routing domain withspecific traffic classes of data capable of being transmitted betweenthe first SDWAN and the second SDWAN; and controlling transmission ofdata between the first SDWAN and the second SDWAN over a specific pathin the segment routing domain based on a traffic class of the data andassociations of the paths with the specific traffic classes of data. 8.The method of claim 7, wherein two nodes in an identified path in thesegment routing domain between the first SDWAN and the second SDWAN areconfigured as path computation element nodes in the WAN underlay by asegment routing controller.
 9. The method of claim 7, furthercomprising: ascertaining performance healths of the paths in the segmentrouting domain between the first SDWAN and the second SDWAN; identifyinga new path in the segment routing domain based on the performancehealths of the paths in the segment routing domain; associating the newpath with a specific traffic class of the data capable of beingtransmitted between the first SDWAN and the second SDWAN; andcontrolling transmission of data of the specific traffic class throughthe new path between the first SDWAN and the second SDWAN based on anassociation of the new path to the specific traffic class.
 10. Themethod of claim 9, wherein the new path is associated with the specifictraffic class to replace a path previously associated with the specifictraffic class.
 11. The method of claim 10, wherein the new path isidentified based on quality of service requirements for transmitting thedata of the specific traffic class between the first SDWAN and thesecond SDWAN.
 12. The method of claim 7, further comprising:ascertaining quality of service requirements for transmitting data of aspecific traffic class between the first SDWAN and the second SDWAN;identifying an appropriate path in the segment routing domain betweenthe first SDWAN and the second SDWAN based on the quality of servicerequirements; and controlling transmission of data of the specifictraffic class over the appropriate path in the segment routing domainbetween the first SDWAN and the second SDWAN through segment routing.13. The method of claim 12, wherein the quality of service requirementsincludes either or both bandwidth and latency requirements fortransmitting data of the specific traffic class.
 14. The method of claim1, wherein the segment routing domain utilizes media access controlsecurity (MACsec) encryption to transmit data between the first SDWANand the second SDWAN.
 15. A system comprising: one or more processors;and at least one computer-readable storage medium having stored thereininstructions which, when executed by the one or more processors, causethe one or more processors to perform operations comprising: identifyinga first software-defined wide area network (SDWAN) and a second SDWAN ofa SDWAN fabric; forming a segment routing domain through the SDWANfabric that interconnects the first SDWAN and the second SDWAN across awide area network (WAN) underlay of the SDWAN fabric by pre-building aplurality of selectable paths through the WAN underlay between the firstSDWAN and the second SDWAN; and controlling data transmission betweenthe first SDWAN and the second SDWAN by performing segment routingthrough the segment routing domain formed between the first SDWAN andthe second SDWAN.
 16. The system of claim 15, wherein the instructionswhich, when executed by the one or more processors, further cause theone or more processors to perform operations comprising: gatheringperformance measurements of links in the WAN underlay forming paths inthe segment routing domain; and performing segment routing through thesegment routing domain based on the performance measurements of thelinks in the WAN underlay to control data transmission between the firstSDWAN and the second SDWAN over the paths in the segment routing domain.17. The system of claim 15, wherein the instructions which, whenexecuted by the one or more processors, further cause the one or moreprocessors to perform operations comprising: identifying paths in thesegment routing domain between the first SDWAN and the second SDWAN;associating the paths in the segment routing domain with specifictraffic classes of data capable of being transmitted between the firstSDWAN and the second SDWAN; and controlling transmission of data betweenthe first SDWAN and the second SDWAN over a specific path in the segmentrouting domain based on a traffic class of the data and associations ofthe paths with the specific traffic classes of data.
 18. The system ofclaim 17, wherein the instructions which, when executed by the one ormore processors, further cause the one or more processors to performoperations comprising: ascertaining performance healths of the paths inthe segment routing domain between the first SDWAN and the second SDWAN;identifying a new path in the segment routing domain based on theperformance healths of the paths in the segment routing domain;associating the new path with a specific traffic class of the datacapable of being transmitted between the first SDWAN and the secondSDWAN; and controlling transmission of data of the specific trafficclass through the new path between the first SDWAN and the second SDWANbased on an association of the new path to the specific traffic class.19. The system of claim 17, wherein the instructions which, whenexecuted by the one or more processors, further cause the one or moreprocessors to perform operations comprising: ascertaining quality ofservice requirements for transmitting data of a specific traffic classbetween the first SDWAN and the second SDWAN; identifying an appropriatepath in the segment routing domain between the first SDWAN and thesecond SDWAN based on the quality of service requirements; andcontrolling transmission of data of the specific traffic class over theappropriate path in the segment routing domain between the first SDWANand the second SDWAN through segment routing.
 20. A non-transitorycomputer-readable storage medium having stored therein instructionswhich, when executed by a processor, cause the processor to performoperations comprising: identifying a first software-defined wide areanetwork (SDWAN) and a second SDWAN of a SDWAN fabric; forming a segmentrouting domain through the SDWAN fabric that interconnects the firstSDWAN and the second SDWAN across a wide area network (WAN) underlay ofthe SDWAN fabric; and controlling data transmission between the firstSDWAN and the second SDWAN by performing segment routing through one ormore changeable paths in the segment routing domain formed between thefirst SDWAN and the second SDWAN.